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Abstract 

Let p, q positive integers. The groups U P (C) and U P (C) x U g (C) act 
on the Heisenberg group H p>q := M P) g(Q) xR canonically as groups of 
automorphisms where M Pt q{G) is the vector space of all complex px q- 
matrices. The associated orbit spaces may be identified with Hj x R 
and S 9 x R respectively with the cone U q of positive semidefinite 
matrices and the Weyl chamber E q = {x S R 9 : x\ > . . . > x q > 0}. 

In this paper we compute the associated convolutions on Tl q x R 
and 5, x R explicitly depending on p. Moreover, we extend these 
convolutions by analytic continuation to series of convolution struc- 
tures for arbitrary parameters p > 2q — 1. This leads for q > 2 to 
continuous series of noncommutative hypergroups on Tl q x R and 
commutative hypergroups on H 9 x R. In the latter case, we describe 
the dual space in terms of multivariate Laguerre and Bessel functions 
on U q and E q . In particular, we give a non-positive product formula 
for these Laguerre functions on S ? . 

The paper extends the known case q = 1 due to Koornwinder, 
Trimeche, and others as well as the group case with integers p due to 
Faraut, Benson, Jenkins, Ratcliff, and others. Moreover, it is closely 
related to product formulas for multivariate Bessel and other hyper- 
geometric functions of Rosier. 

2010 Mathematics Subject Classification: Primary 43A62; Secondary 33C67. 33C52, 
43A90, 43A20 

Key words and phrases: Heisenberg convolution, matrix cones, Weyl chambers, mul- 
tivariate Laguerre polynomials, multivariate Bessel functions, product formulas, hyper- 
groups, hypergroup characters. 
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1 Introduction 

For positive integers p > q consider the vector space M PtQ of all p x q matrices 
over C. Consider the associated Heisenberg group H PA := M PA x R with the 
product 



where tr denotes the trace of the q x q matrix x*y. Clearly, the unitary 
groups K := U p := U P (Q) and K := U p x U q act on H pq via 



for u G U p , v G U q , x G M p>q , and a G R respectively as groups of auto- 
morphisms. The associated orbit spaces may be identified with Tl q x R and 
S ? x R respectively with the cone n g of complex positive semidefinite ma- 
trices and the Weyl chamber E q = {x G R 9 : Xi > . . . > x q > 0} of type B. 
It is well-known that the Banach-*- algebras Mb(H Ptq , K) of all K- invariant 
bounded regular Borel measures with the convolution as multiplication are 
commutative always for K := U p x U q , and for K := U p for q = 1 only (in 
which case the cases K := U p := U P (C) and K := U p x U\ lead to the same 
result). Moreover, in these Gelfand pair cases, the associated spherical func- 
tions are well-known in terms of multivariate Laguerre and Bessel functions; 
we refer to [BJRlj . |BJR2j . jBJRWj . P, jP], jK^], |W] and references there 
for this topic. 

In this paper we compute the associated orbit convolutions onII g xR and 
S g x R explicitly depending on the dimension parameter p. This computation 
is similar to that of Rosier |R2j where the action of U p and U p x U q on M p g 
is considered for the fields R, C, H, and where multivariate Bessel functions 
appear as spherical functions. Moreover, following |R2j . we extend these 
convolutions by analytic continuation to series of convolution structures for 
arbitrary parameters p > 2q — 1 by using the famous theorem of Carleson. 
This extension leads for q > 2 to continuous series of noncommutative 
hypergroups on Ilg x R and continuous series of commutative hypergroups 
on x R. In the latter case, we shall describe the dual spaces in terms 
of multivariate Laguerre and Bessel functions on U q and S g . Moreover, we 
determine further data of these hypergroups like the Haar measures, the 
Plancherel measures, automorphisms and subhypergroups. 

The main results will be as follows: For the (for q > 2 noncommutative) 
hypergroup structures on Il g x R we shall derive in Section 2: 

1.1 Theorem. Let q > 1 be an integer and p G]2g — 1, oof. Define a con- 
volution of point measures onU q x R by 



(x, a) ■ (y,b) = (x + y,a + b — Im tr(x*y)) 



u(x, a) := (ux, a) 



and (u, v )(x, a) := {uxv *, a) 



(8(r,a) *p,q 8(s,b))(f) 



(1.1) 




■ A(I q -w*w) p - 2q dw 
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for f G Cb(Jl q x R), r,s G Tl q , a,b G R, where 

B q := {w G Mq iq : w*w < I q , i.e., I q — w*w is positive definite}, 

A is the determinant of a q x q matrix, and n p <q > is a suitable constant. 
Then this formula establishes by unique bilinear, weakly continuous exten- 
sion an associative convolution on M b (Jl q x R). More precisely, (H q x R, 
is a hypergroup in the sense of Jewett (see [BH\j, f^) with (0,0) as identity 
and with the involution (r,a) := (r, —a). Moreover, 



defines a left and right Haar measure. 

For the commutative hypergroup structures on S g x R we shall derive in 
Section 3: 

1.2 Theorem. Let q > 1 be an integer and p G]2g — 1, oo[. Then S g x R 
carries a commutative hypergroup structure with convolution 



for f G Cfe(Sq x R), (£, a), (r/, b) G S g x R where du means integration with 
respect to the normalized Haar measure on U q and £ G H is identified with the 
associated diagonal matrix in Ii q . The neutral element of this hypergroup is 
given by (0,0) G S g x R, and the involution by (£, a) := (£, —a). Moreover, 
a Haar measure is given by dCj p ^ q (^,a) := h P}q (^) <i£ da with the Lebesgue 
density 



Moreover, the dual spaces of these commutative hypergroups, i.e., the 
sets of all bounded continuous multiplicative functions, will be described 
precisely as a Heisenberg fan consisting of multivariate Laguerre and Bessel 
functions which were studied in |BFj . [F] , [FKj . |Hej and many others. As 
already noticed above, this description is well known for the group cases 
with integer p by [BJRWj , [F] . 

In Section 4 we shall use the product formula on 5, x R of Section 3 in 
order to derive a product formula for the normalized Laguerre functions 
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q^/2) 
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for p > 2q — 1 which are introduced, for instance, in TKj . We shall show 
that for all partitions m and all £, rj G 5 ? , 

JB q JU q V 7 
• e -^H^urju*) A ( Iq _ w * w )P'^dudw. (1.4) 

For g = 1, this formula was derived by Koornwinder [Koj who also gave 
another version of this product formula using Bessel functions. 

We here notice that on all three levels discussed above also degenerated 
product formulas are available for the limit case p = 2q — 1. We do not 
consider the case p < 2q — 1. 

It is a pleasure to thank Margit Rosier for many valuable discussions 
about their multivariate Bessel convolutions and multivariate special func- 
tions. 



2 Heisenberg convolutions associated with ma- 
trix cones 

For positive integers p, q consider the vector space M p>q of all p x q matrices 
over C. Consider the associated Heisenberg group H pq := M Piq x R with the 
product 

(x, a) ■ (y,b) = (x + y,a + b — Im tr(x*y)) 

where tr denotes the trace of the q x q matrix x*y. Clearly, the unitary 
group Up := U P (Q) acts on H VA via 

u(x, a) := (ux, a) for u G U p , x G M Pj(? , a G R 

as a group of automorphisms. We regard U p as a compact subgroup of the 
associated semidirect product G PA := U p k H pq in the natural way and 
consider the double coset space G PtQ / jU p which may be also regarded as 
the space of all orbits of the action of U p on H Pyq in the canonical way. 
Moreover, using uniqueness of polar decomposition oipxq matrices, we see 
immediately that we may identify this space of orbits also with the space 
Ilg x R with 

Ilg := {z G M qtq : z Hermitian and positive semidefinite} 

via 

U p ((x,a)) ~ (\x\,a), 

where \x\ := \/x*x G II g stands for the unique positive semidefinite square 
root of x*x G H q . 

Now consider the Banach-*-algebra Mb(G Ptq \ \U p ) of all f/ p -biinvariant 
bounded signed Borel measures on G Ihq with the usual convolution of mea- 
sures as multiplication. If we extend the canonical projection P : G Ptq — > 
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Gp,q/ /Up — II 9 x R to measures by taking images of measures w.r.t. P, this 
extension becomes an isometric isomorphism between the Banach spaces 
Mb(Gp tq \\Up) and Mb(H q x R). We thus may transfer the Banach-*- algebra 
structure on Mb(G Ptq \\U p ) to M&(II 9 x R) by this isomorphism and obtain a 
probability preserving, weakly continuous convolution * p>q on Mb(Jl q x R) 
in this way. The pair (U q x R, * p q ) forms a so-called hypergroup; for general 
details on hypergroups the the construction above via double cosets and 
orbits we refer to |BH| and [Jj. 

Clearly, this Heisenberg-type convolution * p>q on (measures on) H q x 
R is commutative iff so is Mf,(G P)g \\U p ) , i.e., iff (G Ptq ,U p ) is a Gelfand 
pair. As Gelfand pairs associated with Heisenberg groups were classified 
completely (sec [BJR2], [C], |Kac] . [WJ), it turns out that the convolution 
* P} q is commutative precisely for q — 1. Moreover, for q — 1, the convolutions 
* Ptq on 111 x R = [0, oo[xR and the associated hypergroup structures were 
investigated by several authors; see [Koj . and the monographs [Tj, [BH] as 
well as references therein. 

We next compute the convolution * pA for arbitrary positive integers q 
under the technical restriction p > 2q which will become clear below. We 
do this by using the approach for the Gelfand pair (U p X M p<q , U p ) in [R2j 
where the double coset space (U p x M Ptq / /U p ) is identified with U q , and 
where the same restriction appears. The computation here is only slightly 
more involved, and we obtain: 

2.1 Proposition. Let p >2q>l be integers. Then the convolution of 
point measures is given by 



for f G Cb(rig x R), r, s G Tl q , a,b G R with I q G M q ^ q the identity matrix, 

B q := {w G M qq : w*w < I q , i.e., I q — w*w is positive definite}, 
dw denoting integration w.r.t. Lebesgue measure on M q ^ q , 
A denoting the determinant of a q x q matrix, and 



Proof. The canonical projection ip : H PA — > H p>q ~II,xR from the Heisen- 
berg group onto its orbit space is given explicitly by (p(x,a) = (\x\,a) with 
|x| := yx*x. Moreover, if we define the block matrix 



(<V,a) *p,q S( s ,b)){f) 



(2.1) 




• A(I q - w*w) p - 2q dw 
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an "orbit" (r, a) 6 Il 5 x R has the representative (<J r, a) G H pq . By the 
general definition of the orbit convolution * m (see Section 8.2 of [J] or [R2j ) 
we have 

(S(r,a) *p,q 8(s,b))(f) = ($U p (a r,a) *p,q ^U p (a s,b))(f) 

f(<p((vor, a) ■ u((a s, &))) du 



= I f (\cr r + ucr s\, a + b — Imtr(raQua s)) du (2.2) 
Ju p 

where du denotes integration w.r.t. the normalized Haar measure on U p . 
Using the definition of the absolute value of a matrix above and denoting 
the upper q x q block of u by u q := o-qUo-q G M m , we readily obtain 

(5( r ,a)*p,q8(s,b))(f) = f{Jr 2 + s 2 + ru q s + {ru q s)*, a+6-Im tr(ru q s))du. 
Ju p v 

The truncation lemma 2.1 of [R3j now implies the proposition. □ 

2.2 Remarks. (1) The integral in Eq. ( 12. ip exists precisely for exponents 
p — 2q > —1 which shows that a formula for * M of the above kind is 
available precisely for p > 2q. 

(2) Let p > 2q > 1 be integers, and let / G C b (Il q xR), r, s G Tl q , a, b G R. 
Formula (12. ip and a straightforward computation yield that 

(8(s,b) *p,q $(r,a)){f) 



k PiQ I f (>y r 2 + s 2 + rws + (rws)*, a + b + Im tr(rws)) 

JB q 

■ A(I q -w*w) p - 2q dw. 



If one compares this with Eq. (12. ip . the reader can check directly the 
known fact that * p q is non-commutative precisely for q > 2. For this, 

take for instance, a = 6 = 0, r = ( I °. V and . = ( °) with 
the zero matrix G M g _i )g _i. 
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We next extend the definition of the Heisenberg convolution in Eq. ( 12. ip 
to noninteger exponents p G]2g — 1, oof for a fixed dimension parameter q 
by Carlson's theorem on analytic continuation. For the convenience of the 
reader we recapitulate this result from [Ti], p. 186: 



2.3 Theorem. Let f(z) be holomorphic in a neighbourhood of {z G C : 
Re z > 0} satisfying f(z) = 0(e c ' 2: ') on Re z > for some c < ir. If f(z) = 
for all nonnegative integers z, then f is identically zero for Re z > 0. 

This theorem will lead to the following extended convolution: 
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2.4 Theorem. Let q > 1 be an integer and p G]2g — l,oo[. Define the 
convolution of point measures onU q x R by 

{8(r,a)*p,q8(s,b)){f) (2.3) 

= k P)Q I f (a/ V 2 + s 2 + rws + (rws)*, a + b — Im tr(rws)) 

/or / G Cb(Hq x R) , r, s (z U q , a,b G R where n p<q , dw, A and other data 
are defined as in Proposition ^. 1\ above. Then this formula defines a weakly 
continuous convolution of point measures on Yi q x R which can be extended 
uniquely in a bilinear, weakly continuous way to a probability preserving, 
weakly continuous, and associative convolution on Mb(H q x R). More pre- 
cisely, (U q x R, * p ^ q ) is a hypergroup with (0, 0) as identity and with the 
involution (r, a) := (r, —a). 

Proof. It is clear from Eq. (12.31) that the mapping 

(II, x R) x (n, xR)4 M b (U q x R), ((r, a), (s, b)) ^ 5^ a ) * p , q 5 {Syb) 

is probability preserving and weakly continuous. It is now standard (see [J]) 
to extend this convolution uniquely in a bilinear and weakly continuous way 
to a probability preserving convolution on Mb(U q x R). 

To prove associativity, it suffices to consider point measures. So let 
r, s, t G H q , a, b, c G R, and / G C&(II 9 ). Then 

$(r,a) *p,q (8(s,b) *p,q 8(t,c))(f) 

= K l, q I I f( H ( r , a ,s,b,t,c;v,w)) 

J B q J Bq 

■ A(I q - v*v) p - 2q A(I q - w*w) p - 2q dvdw =: J(p) 
with a certain argument H independent of p. Similar, 

(<V,a) *p,q £(,,&)) *p,q $(t,c) (/) =: I' (p) 

admits a similar integral representation with some integrand H 1 independent 
of p. The integrals I(p) and I'(p) are well-defined and holomorphic in {p G 
C : Rep > 2q — 1}. Furthermore, we know from the group cases above that 
I(p) = I'(p) for all integers p > 2q. As 

\ K P,q\ — 0(\p\ q ) uniformly in {p G C : Re p > 2q — 1} for p — > oo (2.4) 

(see, for example, Eq. (3.9) of |R2J), we obtain readily that 

f(p) := I(p + 2q-l)- I'(p + 2q-l) = 0(\p\ 2q2 ), 

and Theorem 12.31 ensures that I(p) = I'{p) for all p > 2q — 1. Thus * p>q is 
associative. 
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Finally, it is clear by Eq. (12. 3 p that 5(o,o) is the neutral element. More- 
over, as the support supp(5^ :a ) * PA 5( s ,6)) of our convolution is obviously 
independent of p G]2g — 1, <x>[, all further hypergroup axioms from |BH] or 
[JJ regarding the support of convolution products are obvious, as they are 
valid for the group cases with integer p > 2q. □ 

2.5 Remark. The convolution (12.31) obviously satisfies the following sup- 
port formula: For all (r, a), (s, b) G H q x R, 

SUpp(5(r, a ) *p,q C 

C {(t,c) G Ilg x R : < ||r|| + ||s||, |c| < |a| + \b\ + ||r|| • ||s||} 

with the Euclidean norm ||a;|| := yjtr(x*x). 

We next collect some properties of the hypergroups (H q x R, * M ) for 
p G]2g — l,oo[. We first turn to examples of automorphisms. For this we 
first recapitulate that a homeomorphism T on U q x R is called a hypergroup 
automorphism, if for all (r, a), (s, b) G Tl q x R, 

where the left hand side means the image of the measure under T. 

2.6 Lemma. For all u EU q and A > 0, the mappings 

T U: \(r,a) := (Xuru* , \ 2 a) 
are hypergroup automorphisms on (U q x R, * p ^ q ) . 
Proof. Eq. yields 

( S T u>x (r,a)*p,q$T UiX (s,b))(f) 



= K p ^ q I f (\\J u{r 2 + s 2 + ru*wus + (ru*wus)*)u*, 

JB q 

A 2 (a + b — Im tr(uru*wusu*)) ■ A(/ 9 — w*w) p ~ 2q dw. 

Using tr{ut) = tr(tu), \/utu* = uy/tu* and the substitution v = u*wu, we 
see that this expression is equal to 

K p,q I f{^ u V r2 + s2 + rvs + (rvs)*)u*, 

X 2 (a + b-lm tr(rvs)) ■ A(I q - w*w) p - 2q dw 
= T Ui x(S(r ia ) * P:Q d~( S: b)) (2.5) 

as claimed. □ 
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2.7 Remark. The Bessel hypergroups on the matrix cones U q of [R2] admit 
many more hypergroup automorphisms. In fact, a complete classification 
of all automorphisms on these Bessel hypergroups is given in |V3j . Due 
to the additional term Im tr(rws) in Eq. (I2.3p . most of these hypergroup 
automorphisms on U q cannot be extended to our Heisenberg convolutions. 

We next turn to the (left) Haar measure which is unique up to a multi- 
plicative constant by [J]: 

2.8 Proposition. A left Haar measure of the hypergroup (U q x R, * pq ) is 
given by 

u p , q (f) = [ f(V¥,a)A(r)^drda 

JUgXR 

for a continuous function f G C c (Jl q x R) with compact support and the 
restriction of the Lebesgue measure dr on the vector space of all Hermitian 
q x q matrices. 

Moreover, this left Haar measure is also a right Haar measure. 

Proof. We first recall that the Heisenberg groups H P}Q are unimodular with 
the usual Lebesgue measure dX as Haar measure. Therefore, by general 
results on orbit hypergroups (see e.g. [J]), the image (p(dX) of dX under the 
canonical projection (p : H pp — > U q x R is a left and right Haar measure on 
the hypergroup (Jl q x R, * Ptq ). Moreover, the computation in Section 3.1 of 
[R2j shows that 

(fi(d\)(r, a) = Cp ;q ■ A(r) p ' q dr da G M + (U q x R) 

with a certain known constant c Ptq > 0. This proves the result for integers 
p>2q. 

For the general case we must check that 

/ / {6 {r>a) * p , g 6 (VI , b) ){f)A(sy-«dsdb 
Jn q JR 

= ll ( S (V^) *P, q S(r,a))(f) A(s)*-*d8 db 

Jn q JR 

= 11 f(Vs,b) A{s) p - q dsdb (2.6) 
Jn q JR 

for all / G C c ili q x R), r G Ii q} a G R and p G C with Rep > 2q — 1, 
where we use Eq. (12. 3 p also for the convolution for complex p. Clearly, all 
expressions are analytic in p for fixed f,r,a,q. Moreover, by Eq. (12. 3p . all 
three expressions are bounded by 

C\\f\\ ao K M 'M B ^ 

with some constant C and 



M := sup{A(s) : (s, b) G Tl q x R, supp{8(r, a ) * P , g ^,6)) n SU VV f 0} 
= sup{A(s) : (s,b) G (r, -a) * p , q supp(f)}. 
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Using the estimate (12. 4 j) for k Pi3 and the estimate for the support of con- 
volution products in Remark I2.5[ we obtain that the necessary estimate in 
Carlson's theorem 12.31 holds whenever ||r|| and the support of / are con- 
tained in a sufficiently small neighborhood of (0,0). Therefore, (12 .6p holds 
in this case. 

Finally, if / G C c (H q x R) and r G Il 9 are arbitrary, then we choose 
a sufficiently small scaling parameter A such that Ar and the support of 
fx(s,a) := /(A _1 s, A _2 a) are sufficiently small such that ( 12. 6 p holds for Ar 
and f\. As the scaling map Tj ^ is a hypergroup automorphism, it follows 
readily that ( 12. 6 j) for Ar and f\ is equivalent to (12. 6p for r and /. This 
completes the proof. □ 

2.9 Remark. Eq. (12. 3ft implies that for p > 2q — l and (r, a), (s, 6) G 11, x R 
with positive definite matrices r, s, the convolution product 5^ a ) * ps 5{ s ,b) 
admits a density w.r.t. the Lebesgue measure and hence by the preceding 
proposition w.r.t. the Haar measure of the hypergroup (H q x R, * P)9 ). 

In fact, in the case p > 2q — 1 consider the linear map 

w i— > (r 2 + s 2 + rws + (rws)*, Im tr(rws)) 

from 5q C R 2g to 11° x R C R q2 ~ l which has a Jacobi matrix with maximal 
rank q 2 — 1. As the square root mapping on the interior 11° of U q is a 
diffeomorphism, the claim follows immediately from the convolution ( 12. 3p . 

We next turn to the subhypergroups of (Jl q x R, * p>q ). Recapitulate for 
this that a closed set X C Il 9 x R is called a subhyper group, if for all 
x, y G X, we have x G X and {x} * {y} := supp(5 x * 5 y ) C X. We next 
determine all subhypergroups of (Tl q x R, * P)9 ). We begin with examples of 
subhypergroups. 

2.10 Proposition. Let p > 2q — 1, k G {1, . . . , q], and u E U q . Then 

x k , u := { ( w ( o o ) M *' a ) : f e n/c ' a G ^ 

zs a subhypergroup of (H q x 5, * P)9 ) ; and the mapping 

f 

(r, a) i— )■ \ u \ „ „ I M*,a 







zs a hypergroup isomorphism between the Heisenberg hpergroup {Ilk x * P ,fc) 
and the subhypergroup (Xk iU ,* Pig ). 

Proof. The Xkj q are obviously subhypergroups by Eq. (I2.3p . Moreover, us- 
ing the automorphism T u l of Lemma I2.6[ we see that the Xj. >u are subhy- 
pergroups for arbitrary u G U q . 

In order to check that the subhypergroup X k)U is isomorphic with the 
hypergroup (II fc x R, * P) &), we may assume u — I q without loss of general- 
ity. Here we first consider the group cases with integer p > 2q. Here, the 
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inverse image of under the canonical projection (p : H pq — > Tl q x R 
is given by the subgroup |^g^' a ^' xe ^p,k, a G r| of H pq which is 

isomorphic with H p k and preserved by the action of U p . Thus, the preceding 
construction of the orbit hypergroup structures implies that (X kjU , * pq ) is 
isomorphic with (Hfc x R, * Ptk ) as claimed in this case. Therefore, for integers 
p>2q and all / G C&(II 9 x R) and (r, a), (s, b) G Tl k x R, 



r \ \ * p < qS ( ( s 0\ \(fi = ^( r '°) *p,k$(s,b))(fk) 

o oj' a j vv 

with /jfe(r, a ) := / ^ q o j,Alfwe use the definitions of these convo- 
lutions in Theorem 12.41 for arbitrary p, analytic continuation via Carlson's 
theorem yields in the same way as in the proof of Theorem 12.41 that this 
equation holds for all p > 2q — 1. This completes the proof. □ 

2.11 Remark. It follows immediately from Eq. ( 12. 3p . that X := {0} x R 
is a normal subgroup of (H q x R, * p>q ) isomorphic with (R, +). We now may 
consider the associated quotient hypergroup (n, x R, * M )/X which can 
be identified with obviously in a topological way. Using the definition of 
the quotient convolution (see e.g. |V2j ) as well as Eq. f |2.3|) . this quotient 
convolution on U q is given by 



(S r * S s )(f) = k m [ f{^/r^ + s 2 + rws + {rwsy) ■ A(I q - 

JB a 



w*wf- 2q dw. 



In other words, x R, * p (J )/X is isomorphic with the Bessel hypergroup 
structure on the cone Tl q of [R2j with index p. 

2.12 Lemma. Let p > 2q — 1. Let X be a subhypergroup of (U q x R, * Pjq ) 
which is not contained in the subgroup Xq. Then Xq C X . 

Proof. Consider a subhypergroup X (jL X . Thus there exist r G Tl q \ {0} 
and a G R with (r, a) G X. If we restrict the integration in Eq. f |2.3j) to 
matrices w c = (—1/2 + ci) ■ I q G B q with c G [—• y/3/2, \/3/2], we conclude 
from ( 12. 3p and ^2r 2 + rw c r + rw*r = r that 

{r} x[-VS-tr(r 2 )/2,V3-tr(r 2 )/2] C {(r, a)} * p>(? {(r, -a)} C X. 

Therefore, by (12.3p . there exists e > such that for all x G [— e, +e] we have 
(0,x) G {(r, x)} {(r, — x)} C X. As X is a subgroup isomorphic with 
(R, +), it follows that X C X. □ 

2.13 Proposition. Let X be a subhypergroup of (U q x R,* M ). Then X 
is a subgroup of X or X is equal to one of the subhypergroup s X kjU of 
Proposition \2.1(A 
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Proof. Let X be a subhypergroup which is not contained in X . Then X C 
X by Lemma I2.12[ and we may consider the quotient subhypergroup Xj X 
in the quotient hypergroup (Jl q xR)/X which is isomorphic with the Bessel 
hypergroup of [R2j on the cone U q with parameter p. On the other hand, all 
subhypergroups of the Bessel hypergroup structures on the LT^ were classified 
in Proposition 4.6 of |V3j . As X Q C X, this classification leads immediately 
to the classification above. □ 



2.14 Remark. Let 

B:={yeC q : ||y|| 2 < 1} 



and 



S:={2/eC: IM| 2 = 1}. 



By Lemma 3.6 and Corollary 3.7 of |R2j . the mapping P : B q — )■ B q from 
the direct product B q to the ball B q with 



/ 



lj 



Vq) 



yi 

y 2 (/ ? -yfyi) 1/2 



\ 



(2.7) 



\y q (I q - y^y.-i) 1 / 2 •••(/,- y^) 1 ^ ) 

establishes a diffeomorphism such that the image of the measure 

A(I q - w*w) p - 2g dw 

under P _1 is given by nj=i(l ~~ \\yj\\2) p ~ 9 ~''dyi • • • dy q - Therefore, Eq. (12. 3p 
may be written as 

(8(r,a) *p,q 8(s,b))(f) 



(2.8) 



K 



p.q 



r 2 + s 2 + r 



9 



P(y)s + sP(y)*r, a + b — Im tr(rP(y)s)} 



i=i 



I%'ll2) m ^l---^ 



for p > 2g— 1. Moreover, for p — >■ 2g — 1, this convolution product converges 
weakly to the probability measure S(r >a ) *2q-i,q £ M 1 (TL q x R) with 

($(r,a) *2q-l,q 8(s,b))(f) (2-9) 

= K 2q-i, q / / f(^/r 2 + s 2 + rP{y)s + sP(y)*r, a + 6 — Im tr(rP(y)s)) 

JBi- 1 J S 



9-1 

no 



2\p-g-.j 



dyi . . . dy q _ x da(y q 



where a G M l (S) is the uniform distribution on S and «2 9 -i,g > a suitable 
normalization constant. 

This convolution is obviously weakly continuous and can be extended to 
an associative, weakly continuous, and probability preserving convolution 
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on Mb(U q x R) by Theorem 12.41 and taking the limit above. Moreover, all 
further hypergroup axioms may be also checked readily for ( 12.91) . Finally, 
the measure c^-i,,? defined as in Proposition 12.81 is a Haar measure of this 
hypergroup (Il q x R, *2<j-i,<j), the mappings T Ut \(r,a) := (Xuru* , \ 2 a) are 
also automorphisms here as in Lemma 12.61 and the subsets Xj. jU CII g xR 
defined as in Proposition 12.101 are again subhypergroups. 

3 Heisenberg-type convolutions associated with 
Weyl chambers of type B 

In this section we consider the group U q which acts by Lemma 12.61 as a 
compact group {T u i : u G U q } of automorphisms on the Heisenberg hyper- 
groups (Il q x R, * p , g ). As the orbits of the action of U q on Ii q by conjugation 
are described by the ordered eigenvalues £i > £2 > • • • > £g > of a matrix 
in we may identify the space of all £/g-orbits of (Il q x R, with the 
set E q x R where 

2, := U ^...X/K K*':Ci :>...:> 6, >()}. 

The set E q is a closed Weyl chamber of the hyperoctahedral group B q = 
S q ix Zl which acts on R 9 by permutations of the basis vectors and sign 
changes. In this section we show how the convolutions * p ^ q on U q x R for 
p > 2q — 1 lead to orbit hypergroup convolutions o p g on E q x R by using 
methods of [J] or |R3j . In contrast to the hypergroups (Jl q x R, the 
hypergroups (E q x R, o p 9 ) are always commutative. We shall identify the 
characters of these hypergroups in terms of with multivariate Bessel and 
Laguerre functions associated with the root system B q . 

Let us go into the details. Let q > 1 be an integer and p G [2q — 1, oof. 
In the situation described above, the mapping U q — > E q , rH cr(r), which 
assigns to each matrix r its ordered spectrum cr(r), is continuous, surjective 
and open w.r.t. the standard topologies on both sets. Therefore the orbit 
space (Il q x R) Uq (equipped with the quotient topology) may be identified 
with S 5 x R also in a topological way. We now identify both spaces in the 
obvious way and consider the continuous, surjective and open mapping 

$ : LTg x R E q x R, (r, a) h> (a(r), a) 

which corresponds to the orbit map above. This mapping is a orbital map- 
ping from the hypergroup (: Il g x R, * p i9 ) onto E q x R in the sense of Section 
13 of [J], and it follows readily from Section 13 of [JJ that S,xR carries a 
corresponding orbit hypergroup convolution o p q as follows: For a, b G R and 
t;,r] E E q we choose representatives x, y G II 9 with a(x) = £ and a(y) = r\ 
and put 

<%a) °p,q 5(r, fb ) := $(5( x>a ) *p, q S( y>b )). (3.1) 
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The properties of this hypergroup convolution can now be derived similar 
to Section 4 of |R2j . In particular, we can write down the convolution ( 13. ip 
explicitly. For this, we denote the normalized Haar measure on U q by du, and 
£ G E q will always be identified with the diagonal matrix diag(£i, . . . , £ 9 ) G 
H q without mentioning. 

3.1 Theorem. Let q > 1 be an integer and p G]2g — 1, oof. Then S g x R 
carries a commutative hypergroup structure with the convolution 

(%,a) °p,q 8(r,,b) )(/) (3.2) 
= Kp 5 g / / /( ct(a/£ 2 + w] 2 u* + ^wunu* + ur]u*w*$,), 

JBg JUg ^ 

a + 6 — Im tr^wurju*)^ ■ A(I q — w*w) p ~ 2q du dw 

for f G Cb(S g x 5), (£, a), (?7, &) G S 9 x i?. T7ie neutral element is given by 
(0,0) £ 5 9 x 3, and £/ie involution by (£, a) := (£, —a). 
Moreover, a Haar measure on (H 9 x i?, o Pj? ) is given by 

du)p, q (£, a) := hp >q (£) d£ da 

with the Lebesgue density 

v(0:=ner 29+i n^ 2 -^) 2 - ( 3 - 3 ) 

i=l i<j 



Proof. In view of Section 13 of [J] and Section 4 of |R2] on orbit hypergroups, 
we only have to check the commutativity of o p q as well as the statement 
about the Haar measure. 

We first turn to the commutativity. We first observe that for integers 
p > 2q — 1 by its construction, the hypergroup (5 g x R, o p q ) is isomorphic 
with the orbit hypergroup which appears when the group U p x U q acts on 
the Heisenberg group H pq by (u,v)(x,a) := (uxv*,a) for u G U p , v G U q , 
x G M Pi q and a G R. Moreover, it is well known that 

({U p x C/,) x H m ,U p x 

is a Gelfand pair; see [BJRlj . [Dj, [F], |Kacj . Therefore, o pq is commutative 
for integers p > 2q. The general case can now be proved by analytic con- 
tinuation using Carlson's theorem 12.31 in the same way as in the proof of 
Theorem 12.41 We omit the details. 

On the other hand, we may check commutativity also directly. In fact, 
let a, b G R and G E q . We also regard £,77 as real diagonal matrices 
as described above. We obtain from invariance of spectrum and trace by 



Multidimensional Heisenberg convolutions 



15 



conjugations that 

(8(r,,b) °p,q %,o))(/) 

= K P)Q I I f(a(\/u*T) 2 U + £ 2 + U*T)WU£ + (u*T)Wu£)*), 
JB q JU q V 

a + b — Im tr(u*r)wu£,)j ■ A(I q — w*w) p ~~ 2q dudw. 

Substitution w i-» u) as well as diu = <iw, A(I g — w*w) = A(I q — w*w) > 0, 
c(x T ) = cr(x), tr(x T ) = tr(x), £ = £, and fj = rj imply that this expression 
is equal to 

K P,Q / / / \&(\Ju T rfu + £ 2 + U T 7]WU^ + ^U T W*7]u), 
JB q JU q V 

a + b — Im tr(^u T w*r]u) S j ■ A(I q — w*w) p ~ 2q du dw. 

Using the substitution u h-> m t , which preserves the Haar measure on U q , as 
well as the substitution w i-> u*w*u, which preserves the Lebesgue measure 
on Sq, we obtain that the expression above is equal to the right hand side 
of (13.21) . This completes the direct proof of commutativity. 

We finally turn to the Haar measure. By Section 13 of [J], the Haar 
measure 0J Ptq G M + (E q x R) is just given as the image of the Haar measure 
u Pt q G M + (JX q x R) under the projection <3>. As here the second component 
R is not involved, the computation of this image measure can be carried 
out in the same way as in the corresponding proof for the matrix Bessel 
hypergroups in Theorem 4.1 of [R2j . We therefore omit the details. □ 

3.2 Remarks. (1) For p — 2q — 1 the convolution *2g-i,g on II, x R 
introduced in Section 12.141 can be also transfered to a commutative 
hypergroup convolution o 2g _i >g on E q x R in the same way as above. 
We here omit details. 

(2) As already mentioned in the preceding theorem, the hypergroups (S 9 x 
R, o p q ) are orbit hypergroups associated with the action of U p x U q 
on the Heisenberg group H p q for integers p > 2q. Clearly, one may 
also form the associated orbit hypergroup structures (E q x R, o p q ) 
for all integers p > q, where then the corresponding convolution for 
p — q, q + 1, . . . , 2q — 1 is degenerated and no longer given by (13.21) . 

(3) It is clear by the convolution (13.21) that G := {0} x R is a subgroup of 
(E q x R, o p q ) isomorphic with (R, +). We thus may form the quotient 
hypergroup 

(E q x R)/G := {G ■ (£, a) = (£, R) : (£, a) G ~ q x R} ~ S,. 

Using this natural identification as well as the canonical projection 
^ : E q x R — ► Eg, the quotient convolution is defined by 

^ *p,q $v := ^(^,o) °p,q 
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i.e., 

&° M W) (3-4) 
= k va \ \ f(cr(\/^ 2 + urfu* + £wur]u* + ur/u*w*^)) 

JB q JUq 

■ A(I q - w*w) p - 2q dudw 

for / G Cb(E q ), £,r) G H g . In other words, the quotient hypergroup 
((H g x R)/G,* Pi q) is precisely the Bessel- hypergroup on the Weyl- 
chamber E q as studied in Section 4 of |R2j for the field C, i.e., the 
parameter d = 2 there. 

We next turn to the characters of the commutative hypergroups (5 g x 
R, o P)(? ). For this we first recapitulate some basic notions and facts about 
commutative hypergroups mainly from [J] and [BHj . 

3.3 Some facts and notions on commutative hypergroups. Let (X, *) 

be a commutative hypergroup. Then there is a Haar measure u G M + (X) 
which is unique up to a multiplicative constant. We introduce the dual space 

I:={a6 Cb(X) : <5 X * 5y(a) = a(x)a(y) for all x,y G X} 

and the space of all multiplicative functions 

Xb{X) := {a G Cb(X) : <5 X * 5 y (a) = at(x)ce(y) for all x,y & X}, 

and equip both with the topology of locally uniform convergence. Both 
spaces are locally compact, and for a Gelfand pair (G,K), the space of 
spherical functions corresponds to the space x&PO f° r the double coset 
hypergroup (G/ /K, *). The elements of X are called characters. 
We define the Fourier transform . A : L 1 (X, w) — > Cq(X) with 

/(a) := / • /(x) du(x). 

Jx 

Then there exists a unique Plancherel measure 7r G M + (X) such that the 
Fourier transform becomes an L 2 -isometry, i.e., for all / G L x (X,u) fl 
L 2 (X, uj) we have J x \f\ 2 du = Jx\f\ 2 dn, and the Fourier transform can be 
extended to an isometric isomorphism between L 2 (X, oj) and L 2 (X,tt). 

Different to the case of abelian groups, it may occur that supp tt ^ 
X ^ Xb{X). This is the case for instance for Gelfand pairs associated with 
noncompact semisimple Lie groups. On the other hand, there is a growth 
criterion in hypergroup theory which ensures supp n = X = Xb{X). To 
explain this, take a compact set A C X and define recursively the sets 
by AW = A and A (n+1) = A&* * A« = \J xeAMtVeAW supp(5 x * 5 y ). We say 
that (X, *) has subexponential growth if for all compact sets A G X and all 
c > 1 we have co(A^) = o(c n ) for n — > oo. It was proved in |Vog] and |V1] 
that for each commutative hypergroup (X, *) with subexponential growth, 
supp ix = X = Xb{X). 
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We now return to the hypergroups (E q x R, o pq ). As the Heisenberg 
groups have polynomial growth, the following result is not surprising: 

3.4 Lemma. The hypergroups (E q x R, o pq ) have subexponential growth for 
p>2q-l. 

Proof. We see from (13. 2p that for all (^,a),(rj,b) E E q x R and (r, c) G 
supp(5(^ a ) o p q 5( Vt b)) the first (and thus largest) components of the vectors 
r, £, i] satisfy T\ < £i + f?i and |c| < |a| + |6| + Now let C C E q x R be 
compact. Choose d > such that £i < d and \a\ < d for all (£, a) G C. A 
simple induction shows that then for all n G N and all (r, c) G we have 
< r q < . . . < Ti < nd and |c| < nd+ w( - n ~ 1 ^ • d 2 . As the Haar measure Cb v%q 
has a polynomially growing Lebesgue density by Theorem 13.11 the assertion 
is clear. □ 

By the results of Section 13.31 we obtain: 

3.5 Corollary. The hypergroups (E q x R, o p q ) satisfy suppir = X = Xb(X). 

Consider the canonical projection $ : Hq x R — > S g x R as in the beginning 
of this section. This mapping is an orbital morphism in the sense of [J], and 
we conclude from [Jj: 

3.6 Corollary. Let p > 2q — 1. For each character a of (S g x R,o pq ) ! 
the function a o $ g Cb(Jl q x R) is positive definite on the hypergroup 
(U q x R, * p>q ). 

We next introduce a set of characters of the hypergroups (E q x 
R, ° p ,q)- Later on we shall see that this set in fact consists of all characters. 
This set E Pi9 consists of two disjoint sets and T, p q of functions where 
these functions are described in terms of multivariate Laguerre and Bessel 
functions respectively as discussed in [FK . This is not surprising, as this 
connection is well-known for q = 1 (see the product formula in |Ko] , [T] and 
references cited there) as well as for the group cases with integers p,q > 
1; see [F] and references there. Before going into details, we collect some 
notions and facts from |BF] , [F] , |FK] , and |Kan] . We start with some basic 
notions on multivariate special functions: 

3.7 Spherical polynomials. Let m = (mi, . . . ,m q ) be a partition of 
length q with integers mi > ni2 > . . . > m q > 0. We define its length 
|m| := mi + . . . + m q , the generalized Pochhammer symbol 



for x G R (note that we here always use d — 2 in the notion of [FK]), as 
well as the dimension constant 



(x) m = Y[(x ~j + l) 



(3.5) 



d n 



m 



m 



h(m) 2 
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where h(m) is the product of the hook lengths of m; see pp. 237 of [F] and 
p. 66 of |M2j . Moreover, for partitions m we define the spherical polynomials 

&m(x) = / A m (-UXM _1 )(i'U for X G Mq jq 

Ju q 

where du is the normalized Haar measure of U„, A m is the power function 



A m (x) := A 1 (x) m ^ m2 A 2 (x) m "- m:i • . . . • AJx 



and the Aj(x) are the principal minors of the determinant A(x), see Ch. XI 
of |FKj for details. The $ m are homogeneous of degree |m| and satisfy 
$m(0) = for m ^ 0, $o(0) = 1, and <& m (I q ) = 1 for the identity matrix 
I q G Q q - q . 

We also consider the renormalized, so-called zonal polynomials Z m = 
c m $ m with the constants 



_ (g) m |m| 
in h(m) 2 

This normalization is characterized by 



> 0. (3.6) 



{trxf = Z m (x) for k G N . (3.7) 

|m|=fe 

In fact, the normalization constant be easily derived from ( 13. TP and 

some formulas on pp. 237-239 of [Fj; see also Section XI.5 of [FKJ or [Kan]. 
Clearly, we have Z m (I q ) = c m . 

By construction, the $ m and Z m are invariant under conjugation by U q 
and thus depend only on the eigenvalues of their argument. More precisely, 
for Hermitian x G M 9>g with eigenvalues £ = (£i,..-,£ 9 ) G R 9 , we have 
^m(^) = Cm(0 f° r the Jack polynomials C\ (c.f. |FKj . [R2] ) . They are 
homogeneous of degree |m| and symmetric in their arguments. 

We also introduce the generalized binomial coefficients f™J for partitions 
m, n by the unique expansion 



|n|<|m| 



with the identity matrix I q G C q,q . These Binomial coefficients satisfy ( m J 7^ 
only for n C m, i.e. for rij < m; for z = 1, . . . , q. Moreover, it follows from 
[L] that (™) > 0, and that for integers fc, 



|n|=fc V 7 V 7 
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3.8 Multivariate Laguerre polynomials. According to p. 343 of |FKj 
we define the multivariate Laguerre polynomials 

LP ^y-= E (r)^r- $n( - x) (3 - 9) 

|n[<|m| V J yP>U 

and the associated multivariate Laguerre functions 

C(x) := e-^L^x) (3.10) 

for x G C q ' q . The functions and are also invariant under conjugation 
by U q and may thus be regarded as functions in their eigenvalues, i.e., as 
functions on H g . We shall do this from now on without separate notation. 

With a slight change of notation, these Laguerre polynomials and func- 
tions are also considered by Baker and Forrester [BF] in the context of 
Calogero-Sutherland models and Dunkl operators. In fact, a comparison of 
the notions in |FK] and [BF] shows that our polynomials L^x) defined 
above agree with the Laguerre polynomials |m|! • L p ^ q (x; 1) in the notion of 
Proposition 4.3 of [BF] : 

L p m (x) = |m|! ■ L p m q (x; 1) (in the sense of [BF]). (3.11) 

We next collect some known properties about these Laguerre polynomi- 
als: 

3.9 Lemma. The polynomials L^x) form an orthogonal basis on the Hilbert 
space L 2 (Eg, dfip g) with the measure 

q 

du. Piq (x) := Il^'^r 9 ) ■ JJ(,Xi ~ x if dx - 

i=l i<j 

Moreover, for each partition m, 

(tp(X\ 2 J l m l ! (p)m _ , (p) m h(m) 2 

Q-Cxa Q-\Q)m 



"9 

with the normalization constant 



J ^ 



Finally, L p m (0) = (p) m . 

Proof. For the orthogonality and the normalization we refer to Corollary 
XV.4.3 of [FKJ or Proposition 4.10 of [BF]. As the LP m form a basis of 
all polynomials in q dimensions (use e.g. Proposition 4.3 of [BF] and the 
fact that the Jack polynomials form a basis), the completeness of the sys- 
tem (L£j m can be derived by a classical Fourier argument like in the one- 
dimensional case for Laguerre polynomials. Another possibility here is to 
use results of IdJ]. □ 
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We next turn to multivariate Bessel functions of two arguments £, rj G C 



9. 



3.10 Multivariate Bessel functions. According to Kaneko |Kan] (see 
also Section 2.2 of |R2j ) we put 

j (c v) - y CUQCM 



For rj G S g we now define the functions ip*. G C&(H,j x R) by 



(3.13) 



according to Section 4.2 of |R2j .We denote the set of all ^ with 77 G S g by 

^p,q- 

The multivariate Bessel functions appear as limits of the Laguerre func- 
tions above. For the group case with integers p, this was observed already 
by Faraut [Fj. 

3.11 Lemma. Let p >2q — l, and let (m.k)k be a sequence of partitions 
and (Afc)^ CR\ {0} a sequence with — > and lim^oo A& • m^ = r\ G E q . 
Then, forp >2q—l, 



lim 

k— ¥00 



J p (e 2 /2,r? 2 /2) 



uniformly on compact subsets for £ G S ? . 

Proof. Writing the expansions of the Laguerre polynomials and Bessel func- 
tions above in terms of so-called shifted Schur functions precisely as on pp. 
240-241 of [Fj, it can be checked as in Proposition 3.3 of [F] that for all 
partitions n, the coefficients of the expansion of Lm ^ A ^ ) tend to the cor- 

responding coefficients of J p (£ 2 /2, rf/2). Moreover, as <3> n is homogeneous 
of degree |n| with |$ n (a;)| < 1 for ||x|| 2 = 1, and as 

(p)n > (q)m ■ ■ ■ (q)n q > ((g)un|/gj) <? for p > 2q - 1, 
we obtain with Eq. (I3.8P that 



L p mk (0) 



< 



< 



< 



E 



ni/, 
n 



E E (?) 



l<fn(£ 2 ) 
1 



n J ((q) 



|A fc | |n| 
|A,P||£|| 2j 



E 



(|m fe ||A A 



j! • ((<?) 



2j 



< 



OO. 



locally uniformly for £ G H. This readily implies the claimed locally uniform 
convergence. □ 
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Let us return to the characters of the hypergroups (H g x R, o Pi? ). 

3.12 Definition. For A G R\{0} and a partition m, we define the function 

rt,m G C 6 (S, x R) by 

„J» ft ^ ._ p *Ai . *m(W£ 2 /2) _ iXt . -|A|«?+... 5, 2 )/2 . L m(l A l£ 2 ) /o 14 N 

^' J ' &(0) " 6 L p m (0) ^' i4j 

with £ 2 := (£ 2 , . . . , £ 2 ). We denote the set of all (p p Xm with A G R \ {0} and 
partitions m by . 

We notice that for integers p, the functions in the set E p agree with the 
spherical functions <^(A, m, ., .) of pp. 238-241 of Faraut [F] where there the 
Laguerre polynomials are defined with some parameter shift. 

Furthermore, we denote the set of all Bessel functions ^ with n G E q by 
£ pg . Again, for integers p > 1, the set £ pi9 consists of spherical functions 
by [Fj. We note that Faraut (p. 241 of [Fj) uses a slightly different notion 
for these Bessel functions; in his notion we have 

^{r ] ^) = J p {i\r ] ) = ^(V2-i,t) (t G R arbitrary). 



2,i 

3.13 Theorem. Let p > 2q — 1. T/ien a// functions in £ Pj(J := S^UE^ 
are characters of the hypergroup (E q x 5, o P)5 ) . 

Proof. By Corollary 13.51 it suffices to show that all functions in S P)9 are 
multiplicative and bounded. Taking Remark l3.2( 3) and the results of Section 
4 of |R2] into account, this is clear for all functions in Y? . 

The proof for the Laguerre functions in is slightly more involved. 
Here we first consider the group cases with integers p > 2q — 1 where (E 9 x 
R, o p q ) is isomorphic with the orbit hypergroup which appears when the 
group U p x U q acts on the Heisenberg group H p g . In this cases it is well known 
that the functions in T, p correspond to bounded spherical functions on the 
associated Gelfand pairs; see [F] and references cited there. For general 
parameters p > 2q — 1 we again employ analytic continuation by Carlson's 
theorem 12.31 For this we fix a partition m, A G R x , and (£, a), (77, 6) G H g x R, 
and consider the function 



/ / L P\m{ a (V^ 2 + urfu* + ^wur/u* + unu*w*^), 

JBg JUg ' ^ 

a + 6 — Im tr(£wurju*) ) ■ A(/ g — w*w) p ~ 2g du dw 



which has zeros for integer values p > 2q. Moreover, F is analytic on W 
{p G C : Rep > 2g — 1}. Furthermore, because of 

\(pU = \Y[tP-j+i) mj \ > 1 
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for p G W and the definition of the Laguerre functions (p p x m above, we see 
that ip\ m remains bounded for p £ W locally uniformly for (£, a) 6 H 9 x R. 
Therefore, for a suitable constant C > 0, 

\F(p)\ < C 2 + C- \K p>q \ [ [ \A(I q -w*w) p - 2q \dudw <C 2 + C 2 

JU q 

with | A€ P)9 1 = Odpl 92 ) by ( 12.41) for p G W. Therefore, by Theorem 12.31 
F(p) = for all p > 2q — 1 which proves that all elements of are 
multiplicative for all p > 2q — 1. We finally note that the Laguerre functions 
in Ep are obviously bounded by their definition. □ 

As bounded multiplicative functions a on a commutative hypergroup 
satisfy ||a||oo = 1, we obtain: 

3.14 Corollary. For all p > 2q — 1, A G R \ {0} and all partitions m, 
11 p 11 1 

Kmlloc = L 

We next turn to the Plancherel measures ir PA of the hypergroups (5 3 x 
R, o Pi9 ). These measures are well-known for the group cases with integers p 
by |B JRW] . |BJR2j . [F] , as well as for g = 1 in the general case; see e.g. 
Section 8.1 of jT]. 

3.15 Theorem. Leip > 2q — 1. If the hypergroup (5 3 x R,o pq ) is equipped 
with the Haar measure uj p>q according to Theorem \3.1l then the associated 
Plancherel measure Ti p>q G M + (£ PilJ ) according to Section HOI given &?/ 

/ \ 2 9 -g! /" (p) m (g) m , Uhw „ 

/org G L 1 (S p ,j, 7T Pi9 ) where clearly the set £ P)9 of functions is identified with 
its describing parameter set in the obvious way. 

Proof. Let / G C c (S g x R). The classical Plancherel formula for (R, +) says 
that the classical Fourier transform 



F(x,X) := I f(x,t) e~ iXt dt 
./R 



w.r.t. the variable t satisfies 

1 



F(.c,X)\ 2 dXjh Pjq (x) (I.r. 



For fixed A G R\ {0}, we now consider the renormalized Laguerre functions 
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which satisfy (f P \ m {z,t) = e iXt <p? X : m {z). We then obtain with the Haar den- 
sity (I3.3p . the transformation formula, and h Piq (cx) = c 2pq ~ q ■ h p ,g( x ) for 
c > that 

f 1 f V (z 2 /2) 2 

CA,m := \^\,M\ 2 KM dx = j-^ J_ m ^ Q y KM dz - ( 3 - 15 ) 

Moreover, according to Lemma 13.91 the functions (p\ m / ^/C\^ m form an or- 
thonormal basis of L 2 (E q , h PtQ (x) dx). Therefore, by Parseval's identity, 



2 

m>0 </rt 



F(x,X)^ xlm (x) ■ KM dx 



2 

■ — d\ (3.16) 



As by Section 13.31 and Theorem 13.11 the hypergroup Fourier transform / is 
given by 



/(A, m) = ( P\x\,m( X ) ■ *) rft Vafa) dx 

F ( x ^)0\\\, m ( x ) ■ KM dx > 



we obtain from Lemma 13.91 that 

4 



2 

2,Wp,g 

mX)" 1 -' 



?Jr'^ 



2 

2,7r„ 



with the measure 7r p (? introduced in the theorem. As C c (E q x R) is dense 
in L 2 (E q x K,h Pt q(x) dx dt), the equation holds also for all / G L 2 (E q x 
R, h Piq (x) dx dt), which caracterizes the Plancherel measure as claimed. □ 

We next prove that the characters in E p g form the complete dual space 
X of (E q x R, o p q). The following representation of the topology on X as 
a Heisenberg fan is due to J. Faraut who considered the group case with 
integer p: 

3.16 Theorem. Let p > 2q — 1. Then the set £ Pj q is equal to the complete 
dual space X of the hypergroup (E q x R, o p q ) . More precisely, if the closed 
subset 

D := ({0} x E q ) U {(A, A ■ m) : A G R\ {0}, m a partition} 

of R q+1 carries the usual topology as a subset of R q+ , and if X — £ P)(2 
carries the topology of locally uniform convergence, then the mapping 

E : £ p , g D 
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with 

^A,m G K,q ^ (A, Ami, . . . , Am,) 

and 

< e (0,Pi,...,p g ) 
establishes an homeomorphism. 

Proof. We first notice that it follows easily from the definition of the func- 
tions tp\ m and ip p p that these functions on the hypergroup X := E q x R are 
different for different indices. 

We now prove that S p (3 is equal to the complete dual space X. For this 
consider the subgroup G := {0} x R of X as above and the associated 
annihilator 

A(X,Gf) := {a G X : a\ G = 1}. 

This set is a closed subset of the dual X and can by |V2j be identified with 
the dual space of the quotient hypergroup (E q x R)/G which was studied 
in Remark 13727 3). In fact, this hypergroup (E q x R)/£? is (isomorphic to) a 
Bessel-type hypergroup on the Weyl chamber E q as studied in section 4 of 
|R2] . We thus conclude readily from Section 4 of |R2] and the general results 
on annihilators in |V2] that A(X, G) = T^ q holds, and that the mapping E 
above restricted to is an homeomorphism. 

We next turn to the complete dual X. Here we first conclude from Theo- 
rem !3.15l and Corollary 13. 5l that the dual X is the closure of in Cb(E 9 x R) 
w.r.t. the topology of locally uniform convergence. To prove that this clo- 
sure is equal to £ Pi9 , we consider some a G X C Cb(E q x R) which is the 
locally uniform limit of a sequence (<fx n mn )n>i C . As the restrictions 
to G also converge locally uniformly, it follows from the definition of the 
V^Am (An)n>i C R \ {0} converges to some A G R. We consider two 
cases: 

(1) If A = 0, then it follows from the definition of the <p p Xm that a G 
A(X,G), and the preceding consideration implies that a G . 

(2) Let A 7^ 0. If (m n )„>i remains bounded, we may choose it as a constant 
sequence with m„ = m without loss of generality, and we obtain from 
A n — > A that a = lim n </?^ m = v^A.m e ^p,q- ^ e ^ nus ma y res trict our 
attention to the case where at least one component of m n tends to 
infinity. In this case we take an arbitrary multiindex m and conclude 
from the convergence of normalized Laguerre polynomials L\ m to 
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L p x m (see Section 13 .8 j) that for the density h pq of (13. 3p 



SSL, K,m»( 2 ' ) " ^L,m(^°))V<^) dz 

^„(|A„k 2 /2)^(|A„k 2 /2) 



— n— >oo 



J=. q n ^°° L^„(0)L^(0) 

Using a renormalization as in Eq. (I3.15P with A n — > A, the dominated 
convergence theorem, and the fact that the modulus of the tpK „ is 
bounded by 1, we obtain that that this expression is equal to 

1 f i7 m ,> 2 /2)L^ 2 /2) _ |N|2 



\\\™ n->oo J Bq L p mn (0)L p m {0) 

Therefore, as a G Cb{E q x {0}) C L 2 (S 9 , e _A " z " 2 /i Pi(? (z)c?z), and as the 
Laguerre polynomials (L p Xm (z 2 )) m with squared arguments form an 
orthogonal basis of this L 2 -space by Lemma I3.9[ it follows that a = 
a.s. on S g x {0} and thus on S g x R. This is a contradiction of a being 
a continuous character with a(0, 0) = 1. 

Summarizing, we conclude that E p j(? = X, and that a sequence (ip p x m ) n >i C 
Sp can converge to a character without loss of generality only in the follow- 
ing two cases: Either A n -> A ^ and (m„) n is finally constant or A„, — > 0. 
In the first case, locally uniform convergence obviously appears, and in the 
second case we have locally uniform convergence for A„m„ — > r\ G E q by 
Lemma 13.111 

We finally prove that a sequence (^P P \ n mn )n>i C £p )(3 can converge lo- 
cally uniformly to some ip p only for parameters with lim n A n m n = p. If 
this is done, it follows readily from our preceding informations that the 
mapping E of our theorem is an homeomorphism. In order to prove this 
statement, consider such a sequence (ip^ mn )n>i and its limit ip p p . If the se- 
quence (A n m n ) n C E q is bounded, we find a convergent subsequence with 
some limit p G E q . For this subsequence we obtain ip p Xn mn — > locally 
uniformly, and thus ijjp = ?p p , i.e., p = p. Therefore, each convergent sub- 
sequence of the bounded sequence (A n m n ) n C E q converges to p which 
implies A„m n — >• p as claimed. We finally consider the unbounded case. 
Here we may assume without loss of generality that the largest component 
of (A n m n ) n C S q converges to oo, i.e., that A n mi i?1 — > oo. We then define 
A n := p/nii jn and observe that A n m n has a subsequence which converges to 
some p G E q with pi = 1. For this subsequence and any (£,£) G E q x R we 
the have 
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as well as by the definition of (p p x m and our assumption about locally uniform 
convergence, 

^.-n^'*) = V P X nk ,m nk tt ■ ^KjK k ,t- KJK) ^(0,0) = 1. 

It follows that ip p ~ = I contradicting pi — 1. Therefore this limit case cannot 
appear which completes the proof of the theorem. □ 

The proof of the continuity of the mapping E above is quite special. 
It was pointed out to the author by J. Faraut that a more systematic ap- 
proach for this part of the theorem is available by using heat kernels on 
the hypergroups S,xR and their explicit continuous hypergroup Fourier 
transfoms. 

We finally turn to a refinement of the last part of the theorem. The 
following observation is clear from Section 2: 

3.17 Lemma. Let (£, a), (77, b) G S g x R with £ q > and n q > 0. Then, for 
p > 2q — 1, the convolution product S^ >a ) o pq S( Vt fy is absolutely continuous 
w.r.t. the Haar measure uJ p ,q- 

Proof. As absolute continuity is preserved under the continuous projection 
$ : U q x R — > H g x R, the lemma follows immediately from Remark 12.91 □ 

As by the Riemann-Lebesgue Lemma for hypergroups (see [J]) the hyper- 
group Fourier transform maps functions in L l (E, q x R, £j p>q ) to Co-functions 
on the dual space X, the preceding lemma and Theorem 13.161 imply the 
following. 

3.18 Corollary. Let (£, a) GH,x R with £ q > 0. Then, forp > 2q — 1, the 
hypergroup Fourier transform of <%, a ) °n the dual space X is a Co-function. 
In particular, for (eH, with £ q > 0, 

4 A product formula for Laguerre functions 

In this section we derive a product formula for the multivariate Laguerre 
functions of Section 13.81 for p > 2q — 1. For q = 1, this formula was 
established directly by Koornwinder [Ko] who also discusses its connection 
with Heisenberg groups. We here derive the product formula from the prod- 
uct formula (\3.2h for p > 2q — 1 and its degenerate version for p = 2q — 1 
according to Remark l3~2T l) for the characters tp p Xm G of the commuta- 
tive hypergroups (E q x R, o p (? ). We here shall use the general approach of 
[RV] where, embedded into a more general setting, it is also explained how 
for q = 1 Koornwinder's product formula for the one-dimensional Laguerre 
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functions corresponds to the Heisenberg-type hypergroup convolution on 
[0, oo[xR. We now extend this approach from q — 1 to q > 1. 

For this we recapitulate from Remark l3.2l that G := {0} x R is a subgroup 
of the commutative hypergroup (E q x R, o p (? ) for p > 2q — 1. Moreover, 
r(x,t) := e lt defines a function r G Cb(E q x R) with 



\r(x, t)\ = l, r((x,t)) — r(x,t), and r((x, t) ■ (0, s)) — r(x, t) ■ r(0, s) 

for all x £ E q and s, t G R. In other words, r is a partial character of 
(E q x R, o p q ) with respect to G according to Definition 4.1 of |RVj . We now 
consider the canonical projection 



p:B,xR->(H,xR)/G~H 



according to Remark [3.2( 3) and recapitulate that the quotient hypergroup 
(E q x R)/G agrees with the corresponding Hermitian Bessel hypergroup 
on E q of Rosier [R2], i.e., the corresponding hypergroup involution is the 
identity mapping. Following Section 4 of [RV], we now define a deformed 
quotient convolution of point measures on E q by 

h *T,p,q S v := P(T ■ (% ) °p,q <W))) fOT £' V e -q- 

According to Section 4 of |RVj . this convolution can be uniquely extended 
in a weakly continuous bilinear way to a commutative Banach-*-algebra 
(Mb(E q ), m T>p q ) with the total variation norm as norm. More precisely, by 
Theorem 4.6 and Corollary 4.7 of [RVj . (E q ,m TtM ) becomes a Hermitian 
signed hypergroup in the sense of [Rlj : see also |RV] and |Ross] for the 
notion of signed hypergroups. 

Let us compute the convolution (14. ip in an explicit way. Eq. (13. 2p for 
p > 2q — 1 shows that for / G Cb(E q ) we have 

S t 9 T,P,q W) = f M T ■ (%0) °p,q S(r,,0))) 

JXia 

f(x) ■ e lt rf(% i0 ) o Pi9 5 {ri>0) )(x, t) 

= K pq / / /(ct(a/^ 2 + W] 2 U* + ^WUr/U* + U7]U*W*^)j 
JB q JU q ^ ' 

■ e-i-^HSvunu*) . A ^ _ w * w y-^dudw. (4.2) 

For p = 2q—l one obtains a corresponding degenerate version of this formula 
by using Remark 13.2( 1) and Section T2.14I We notice that this convolution 
is obviously not probability preserving and usually even not positivity pre- 
serving. 

Moreover it follows from Theorem 5.2 of [RVj that for p > 2q — 1 and all 
partitions m, the normalized Laguerre functions 

~p / \ ._ fmOgVg) _ -(x? + ...+xg)/2 -^m( a;2 ) ( T (zW ) 

Vm[h C(0) LSx(O) 



S,xR 
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form all bounded R- valued multiplicative functions on (H g , • T , p ,g). In sum- 
mary, we have the following product formula: 

4.1 Corollary. For all p > 2q — 1, £, r\ G H g; and a// partitions m, 
^(0-^(17) = 

= / / yj^J Ct(a/^ 2 + U7] 2 U* + ^WUTjU* + UT}U*W*£) 

JB q JU q ^ 

• e -'-tof(^«'7«*) A ( /g _ w * w )P- 2 9d M (4.3) 

Moreover, for p = 2q — 1, 

J J S J Uq 

■ e-^*^^ ■ - \\y 3 \\ 2 2 ) m du dm... dy q -i ds(y q ) 

3=1 

with y := (yi,...,y q ) and B := {y G C 1 : ||w|| 2 < 1}, where s G M 1 (S) 
denotes the uniform distribution on the sphere S := {y G C 9 : ||w|| 2 = 1}, 
and P zs i/ie map defined in \2. 7| ). 



As the convolution • TjP( g of (14.21) is not mass preserving, there does not 
exist an associated translation invariant measure m TtPtq G M + (E q ) in the 
usual hypergroup sense. However, Theorem 4.6 of [RV] ensures that the 
Haar measure 

dm P , q (0 = f[^- 2q+1 life 2 - * e M+ ( S «) 

i=l i<j 

of the usual quotient hypergroup (E q xR)/G ~ S 9 also admits the following 
adjoint relation for • T , P , q ' If for / G C c (E q ) and £ G H g we define the translate 

then we have for all f,gE C c (E q ) 

/ (Ttf) ■ g dm p , q = / (T e #) ■ / dm PS . (4.4) 

4.2 Remark. Consider the group case with integer parameters p > 2q — 1. 
Here, all characters of the double coset hypergroup (E q x R, o pq ) correspond 
to positive definite spherical functions on the Heisenberg group H m and 
admit therefore a dual product formula, i.e., for all a, j5 G T, Pjq (see Theorem 
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13 . 1 6[) there is a unique probability measure p a ^ G M 1 (S p (J ) such that for 
all (x, t) e E q x R, 

a(x,t) ■ /3(x,t) = j(x,t) dp a ^(j). (4.5) 

Let us take a(x,t) := e Ali < ■ C( ^|g /2) and (3(x,t) := e X2it ■ '"%* 2/2) for 
Ai, A 2 7^ and partitions m, n. 

If we consider the t-dependence of the product in H4.5[) for x = 0, we 
obtain for Ai, A 2 > that 

L p m (XiX 2 /2) Ll(\ 2 x 2 /2) 
LL(0) ' L£(0) 

/ . x x v L p k ((Ax + A 2 )^/2) 

[k[<|m|+|n| kl J 

for unique coefficients c(m, n, k; Ai, A 2 ,p, q), which satisfy 

c(m, n, k; Ai, X 2 ,p, q) > and ^ c(m, n, k; Ai, X 2 ,p, q) = 1. 

k 

For instance, for n = 0, we get for < Ai < A 2 that 

I/ m {X x x 2 /2) = ]Tc(m,k;p,g)L£(A 2 x 2 /2) 

k 

with nonnegative coefficients c(m, k.;p,q). 

On the other hand, for Ai > and — Ai < A 2 < we obtain 

^(A^/2) ^ n (|A 2 |x 2 /2) v / P k ((Ai + A 2 )x 2 /2) 

— " v S°) ' 

We expect that these and further related results also hold for arbitrary 
(noninteger) p > 2q — 1. For g = 1, the formulas above are connected with 
the discrete Laguerre convolution in |AG] . 
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